Goto

Collaborating Authors

 Pompano Beach


SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View

Xiao, Yongjie, Liang, Hongru, Qin, Peixin, Zhang, Yao, Lei, Wenqiang

arXiv.org Artificial Intelligence

Despite the great potential of large language models(LLMs) in machine comprehension, it is still disturbing to fully count on them in real-world scenarios. This is probably because there is no rational explanation for whether the comprehension process of LLMs is aligned with that of experts. In this paper, we propose SCOP to carefully examine how LLMs perform during the comprehension process from a cognitive view. Specifically, it is equipped with a systematical definition of five requisite skills during the comprehension process, a strict framework to construct testing data for these skills, and a detailed analysis of advanced open-sourced and closed-sourced LLMs using the testing data. With SCOP, we find that it is still challenging for LLMs to perform an expert-level comprehension process. Even so, we notice that LLMs share some similarities with experts, e.g., performing better at comprehending local information than global information. Further analysis reveals that LLMs can be somewhat unreliable -- they might reach correct answers through flawed comprehension processes. Based on SCOP, we suggest that one direction for improving LLMs is to focus more on the comprehension process, ensuring all comprehension skills are thoroughly developed during training.


Futuristic headset can 'bring the world' to homebound elderly thanks to AI tech

FOX News

MyndVR Chief Operating Officer Dave Rawlins discussed with Fox News Digital his company's goal of creating connectivity through a virtual space without trying to'outsource' care to'a machine.' A company focused on developing virtual reality (VR) therapy treatments will turn to artificial intelligence (AI) to create immersive environments that could revolutionize therapy treatment, especially for the elderly and those with mobility issues. "We want to bring the world to residents who otherwise can't experience the world," MyndVR's Chief Operating Officer Dave Rawlins told Fox News Digital. "A lot of what we do is 360 content, provide experiences and applications, modules that give a resident the opportunity to potentially get out of the four walls of the facility, whether that be in travel or Broadway shows or experiencing things that they may have experienced in their earlier life," Rawlins explained. Launched in 2016, MyndVR developed lightweight VR headsets that allow users to engage in gamified environments that help them relax and distract them from a therapy session or boost their moods by leaving their homes to do things they are unable to physically do anymore.